Method of operating electric device including image sensor

ABSTRACT

A method of operating an electronic device includes displaying a preview image in response to execution of a camera application, extracting feature information from the preview image, converting user input to an input value in response to the user input generated on the preview image, setting a depth based on the feature information and the input value, and generating a result image in accordance with the depth in response to execution of an imaging operation.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority to Korean Patent Application No.10-2019-0156527 filed on Nov. 29, 2019 and Korean Patent Application No.10-2019-0064237 filed on May 31, 2019 in the Korean IntellectualProperty Office, the collective subject matter of which is herebyincorporated by reference.

BACKGROUND

Example embodiments of the inventive concept relate generally to methodsof operating an electronic device including an image sensor.

An image sensor is a semiconductor device capable of generatingelectrical signal(s) in response to incident light received by anelectronic device incorporating the image sensor. The image sensor maycooperate with other components to generate an image based on theelectrical signal(s). Various image sensors have been employed in avariety of electronic devices. Recently, in order to improve the qualityof images obtained by an image sensor and to generate images moresatisfying to user demands, various additional functions have been addedto the image sensor. Various applications, executable by the electronicdevice including an image sensor have also been developed.

SUMMARY

An example embodiment of the inventive concept provides a method ofoperating an electronic device which satisfies various user demands bydetermining a depth of a result image based on feature informationobtained from a preview image and/or a user input occurring on a previewimage, and the like.

According to an example embodiment of the inventive concept, a method ofoperating an electronic device includes displaying a preview image inresponse to execution of a camera application, extracting featureinformation from the preview image, converting user input to an inputvalue in response to user input generated on the preview image, andsetting a depth based on the feature information and the input value,and generating a result image in accordance with the depth in responseto execution of an imaging operation.

According to an example embodiment of the present inventive concept, amethod of operating an electronic device includes displaying an apertureicon having an aperture shape on a preview image in response toexecution of a camera application, adjusting an opening area of theaperture icon displayed on the preview image in response to user inputgenerated on the preview image, and adjusting blur strength of abackground displayed on the preview image in response to user input.

According to an example embodiment of the present inventive concept, amethod of operating an electronic device includes performing a learningoperation of a machine learning model by applying feature informationextracted from each of images stored in a memory as input parameters andapplying blur strength extracted from each of the images as outputparameters, extracting the feature information from a preview image inresponse to execution of a camera application, and inputting the featureinformation to the machine learning model, and determining blur strengthof a background displayed on the preview image based on an output valueof the machine learning model.

BRIEF DESCRIPTION OF DRAWINGS

The above and other aspects, features, and advantages of the presentinventive concept will be more clearly understood from the followingdetailed description, taken in conjunction with the accompanyingdrawings, in which:

FIGS. 1A and 1B are respective diagrams illustrating an electronicdevice according to an example embodiment of the present inventiveconcept;

FIG. 2 is a block diagram illustrating an electronic device according toan example embodiment of the inventive concept;

FIG. 3 is a block diagram further illustrating an image sensor accordingto an example embodiment of the inventive concept;

FIG. 4 is a diagram further illustrating a pixel array of an imagesensor according to an example embodiment of the inventive concept;

FIGS. 5 and 6 are respective circuit diagrams illustrating a pixel of animage sensor according to an example embodiment of the inventiveconcept;

FIGS. 7, 8 and 9 are respective flowcharts summarizing methods, ormethod portions, of operating an electronic device according to anexample embodiment of the inventive concept;

FIGS. 10 and 11 are respective diagrams illustrating a method ofoperating an electronic device according to an example embodiment of theinventive concept;

FIG. 12 is a block diagram illustrating operation of an electronicdevice according to an example embodiment of the inventive concept;

FIG. 13 is a chart illustrating data relationships that may be stored ina look-up table used in an electronic device according to an exampleembodiment of the inventive concept;

FIG. 14 is a flowchart summarizing a method of operating an electronicdevice according to an example embodiment of the inventive concept;

FIG. 15 is a diagram further illustrating a method of operating anelectronic device according to an example embodiment of the inventiveconcept;

FIG. 16 is a diagram illustrating an artificial neural network that maybe used in an electronic device according to an example embodiment ofthe inventive concept;

FIG. 17 is a chart listing nodal data from the neural network of FIG.16;

FIG. 18 is a block diagram illustrating an operation of an electronicdevice according to an example embodiment of the inventive concept;

FIG. 19 is a flowchart summarizing a method of operating an electronicdevice according to an example embodiment of the inventive concept; and

FIGS. 20, 21, 22 and 23 are respective diagrams illustrating operationsof an electronic device according to an example embodiment of theinventive concept.

DETAILED DESCRIPTION

Hereinafter, embodiments of the inventive concept will be described withreference to the accompanying drawings.

FIGS. 1A and 1B are diagrams illustrating an electronic device accordingto an example embodiment.

Referring to FIGS. 1A and 1B, an electronic device 1 may be implementedas a mobile device, such as a smartphone. However, the electronic device1 is not limited to only mobile devices, such as a smartphone, butinstead may include other devices providing a camera function forimaging an object.

The electronic device 1 illustrated in FIG. 1A includes a housing 2, adisplay 5 and cameras 6, 7 and 8, among other components. In the exampleembodiment, the display 5 substantially covers a front side of thehousing 2. Here, the display 5 may be visually presented to provide oneor more graphical user interface(s) GUI. For example, the display ofFIG. 1A is shown divided into a first region 3 and a second region 4, Ina manner well recognized by those skilled in the art, a user mayinteract (or manipulate) one or more icons presented by the GUI toprovide user input to the electronics device 1. In response to userinput one or more application(s) loaded to the electronic device 1 maybe selectively executed (or performed).

Referring to FIG. 1A, certain front facing cameras provided by theelectronic device 1 may include a first front facing camera 6 and asecond front facing camera 7, wherein each front facing camera mayoperate according to respective, different characteristics. In thisregard, those skilled in the art will recognize that cameras associatedwith electronic devices according to example embodiments may include anaperture (e.g., a controllable opening through which electromagneticenergy (e.g., visible light) may pass to become incident light on animage sensor). One example of different characteristics noted above isthe degree to which a respective aperture is opened or closed.

As an example, the first front camera 6 and the second front camera 7may have different aperture values, different focal lengths, differentfields of view, and the like. In certain example embodiments, the firstfront camera 6 may be configured as a general camera, and the secondfront camera 7 may be configured as a time-of-flight (ToF) camera. Whenthe second front camera 7 is configured as a ToF camera, it may beoperationally combined with a light source to provide a distancemeasuring function, a depth-map generating function, a face recognitionfunction, and the like.

FIG. 1B illustrates a rear side of the electronic device 1. Here, theelectronic device 1 includes a rear facing camera 8 and a light emittingunit 9. The rear camera 8 may include a plurality of rear cameras (e.g.,rear cameras 8A , 8B and 8C), wherein each rear facing camera mayoperate according to respective, different characteristics (e.g.,aperture value, field of view, number of image sensor pixels, etc.). Thelight emitting unit 9 may include (e.g.,) one or more Light EmittingDiode(s) (LED) as a light source, and may operate as a camera flashelement under the control of one or more application(s) or user input inrelation to the rear camera 8.

With regard to the foregoing, those skilled in the art will understandthat the terms “front” and “rear” are relative in nature and may bearbitrarily defined according to actual implementations of theelectronics device 1.

With regard to the embodiment(s) illustrated in FIGS. 1A and 1B, theelectronic device 1 may be variously implemented with a plurality offront and/or rear facing cameras 6, 7, 8A, 8B and 8C. In view of growingconsumer demand variable and increasingly high quality images, theplurality of front and/or rear cameras 6, 7, 8A, 8B and 8C musttypically be able to operate with different aperture values, differentnumbers of image sensor pixels, different fields of view, differentfocal lengths, etc. Further, greater user control (e.g., imagingfunctions) over the generation and display of images is demanded. Forexample, certain electronic devices 1 provide an imaging functionwherein a user may provide user input in relation to a preview imagedisplayed on the display 5. In response to the user input one or morecamera applications may be executed on the electronic device 1 tovariously adjust the preview image to generate a result image pleasingto the user.

Hereafter, the term “camera” will be generally used to denote theprovision and/or use of one or more front facing and/or rear facingcamera(s).

In certain embodiments, an aperture value of a camera capturing andproviding a preview image may not be directly changed in response to auser input for adjusting an aperture value on the preview image.Instead, the electronic device 1 may execute an image processingapplication in response to user input in order to adjust the aperturevalue with respect to the preview image, such that the blur strength ofa background portion of the preview image is reduced and/or a resultimage is improved relative to the preview image. Accordingly, the resultimage may be generated as if an aperture value of the camera and/or adepth value were changed in response to direct user input. In thisregard, those skilled in the art will recognize that an image may becharacterized by a particular depth. Depth is a term that denotes thevisual perception range into an image. Depth may be referred to as“deep” or “shallow”, but these are merely relative terms and may varyaccording to application or image perception requirements.

In contrast, when a user provides user input with respect to a previewimage in order to adjust the aperture value, the electronic device 1 maycollect feature information, such as composition information regardingan object and its associated background, information regarding anambient environment, and the like. In response to the featureinformation, as well as the user input, a depth of the result image maybe accurately generated. For example, imaging parameters optimizedaccording to feature information and/or user input may be automaticallyset using a pre-stored look-up table and/or a machine learning model.That is, one or more learning operation(s) may be executed with respectto various feature information and/or user input in order to defineappropriate (or user desired) imaging parameters. The imaging parametersset by the electronic device 1 may include a depth corresponding to blurstrength of the background included in the result image, colors of theresult image, etc. In certain embodiments, one or more look-up table(s)and/or machine learning model(s) may be used to generate result frompreview images captured by the electronic device 1 under the control ofthe user, or previously captured and stored images.

FIG. 2 is a general block diagram illustrating an electronic device 10according to an example embodiment.

Referring to FIG. 2, the electronic device 10 may include a display 11,an image sensor 12, a memory 13, a processor 14 and a communicator 15,among other components. Collectively, the elements 11, 12, 13, 14 and 15included in the electronic device 10 may variously transmit and/orreceive data using a bus 16. In this regard, the elements 11, 12, 13,14, 15 and 16 illustrated in FIG. 2 are block level examples presentedhere to provide a descriptive context. The electronic device 10 mayinclude additional elements as will be appreciated by those skilled inthe art.

The image sensor 12 may be disposed on a front side and/or a rear side,or other regions of the electronic device 10 and may provide a camerafunction, as described above with reference to FIGS. 1A and 1B. Theprocessor 14 may be configured as an application processor, a CentralProcessing Unit (CPU), or the like, and may be used to control theoverall operation of the image sensor 12.

In response to user input, one or more camera applications running onthe electronic device 10 may capture a preview image that is thendisplayed on the display 11. The user may then generate a result imagefrom the preview image (e.g.) by touching a shutter icon displayed on auser interface (e.g., a GUI) together with the preview image on thedisplay 11, or by imaging an object using an input unit arranged in thehousing of the electronic device 10. In the example embodiment, a userinterface by which a user may directly adjust blur strength of abackground, and the like, displayed on the preview image may bedisplayed on the display 11 along with the preview image.

When a user provides user input via the user interface, the electronicdevice 10 may determine imaging parameters using various featureinformation (e.g., ambient environment information, backgroundinformation for the preview image, and the like) along with the userinput in order to generate a result image using the determined imagingparameters. As an example, the electronic device 10 may determine theimaging parameters using a look-up table stored in the memory 13 and/ora machine learning model stored, at least in part, in the memory 13.

As an example, the imaging parameters may include color(s), a focusdepth (or depth), and other elements defining the result image. Here, adepth of the result image may change in response to user input providedby a user via the user interface. In the example embodiment, a degree ofan out-focusing effect on a result image may be varied by the userinput. Thus, the electronic device 10 may provide an imaging function,as if a desired depth were defined by directly opening and closing anaperture in a manually operated camera by the user input in response tothe preview image. Also, by changing the user interface, an opening areaof an aperture icon, for example, displayed on the preview image inresponse to the user input generated in relation to the preview image,changes in aperture value which may change a depth of the result imagemay be intuitively transferred to a user.

FIG. 3 is a block diagram illustrating an image sensor 100 according toan example embodiment.

Referring to FIG. 3, the image sensor 100 may include a pixel array 110and a controller 120, wherein the controller 120 may include a rowdriver 121, a read-out circuit 122, a column driver 123 and a controllogic 124, among other components.

The image sensor 100 may be used to convert incident light received bythe electronic device 10 into corresponding electrical signal(s), suchas image data. The pixel array 110 included in the image sensor 100 mayinclude a plurality of pixels PX, wherein each pixel PX in the pluralityof pixels PX includes at least one photoelectric device generatingelectric charges in response to received incident light. Here, thephotoelectric device may be a photodiode (PD), for example. In anexample embodiment, each of the plurality of pixels PC may include twoor more photodiodes. The two or more photodiodes may be included in eachof the plurality of pixels PX for each of the pixels PX to generatepixel signals corresponding to a variety of colors or for the purpose ofproviding an autofocusing function.

Each of the plurality of pixels PX may include a pixel circuit forgenerating a pixel signal from electric charges generated byphotodiodes. As an example, the pixel circuit may include a transfertransistor, a driver transistor, a select transistor, a resettransistor, and the like. The pixel circuit may detect a reset voltageand a pixel voltage from each of the plurality of pixels PX and maycalculate a difference therebetween, thereby obtaining a pixel signal.In the pixel voltage, electric charges generated by the photodiodesincluded in each of the plurality of pixels PX may be reflected. In anexample embodiment, two or more adjacent pixels PX may be included in asingle pixel group, and the two or more adjacent pixels PX included inthe pixel group may share at least a portion of the transfer transistor,the driver transistor, the select transistor, and the reset transistor.

The row driver 121 may drive the pixel array 110 according to row units.For example, the row driver 121 may generate a transfer control signalfor controlling the transfer transistor, a reset control signal forcontrolling the reset transistor, a select control signal forcontrolling the select transistor, and the like, the transfertransistor, the reset transistor, and the select transistor included inthe pixel circuit.

The read-out circuit 122 may include a sampling circuit, ananalog-to-digital converter (ADC), and others. The sample circuit mayinclude a plurality of samplers, and in an example embodiment, thesampler may be configured as a correlated double sampler (CDS). Thesampler may be connected to the pixels PX included in a row lineselected by the row driver 121 through column lines, and may detect areset voltage and a pixel voltage from the respective pixels PX. Thesamplers may compare each of the reset voltage and the pixel voltagewith a ramp voltage, and may output a result of the comparison. Theanalog-to-digital converter may convert the result of the comparisonoutput by the samplers to a digital signal and output the digitalsignal.

The column driver 123 may include a latch which may temporality storethe digital signal, or a buffer circuit and an amplifier circuit, andmay process the digital signal received from the read-out circuit 122.The row driver 121, the read-out circuit 122, and the column driver 123may be controlled by the control logic 124. The control logic 124 mayinclude a timing controller for controlling an operation timing of therow driver 121, the read-out circuit 122, and the column driver 123, animage signal processor for processing image data, and others.

In example embodiments, the image signal processor for processing imagedata may be included in a processor 150. The processor 150 may processimage data and may generate a result image, and may perform operationsof displaying the result image on a display or storing the result imagein a memory.

At least one of the controller 120 and the processor 150 of the imagesensor 100 may change an imaging parameter which may determinecharacteristics of the result image on the basis of a user inputgenerated in relation to the preview image and/or feature information.As an example, the imaging parameter may include blur strengthrepresenting a depth of the result image, and the like. For example,when the blur strength increases, a background other than an arearecognized as an object in the result image may be blurred such that adepth of the result image may become shallow. When the blur strengthdecreases, the background may become clear in the result image such thata depth may become deep.

A method of determining the imaging parameter in response to user inputmay vary with implementation of example embodiments. As an example, animaging parameter corresponding to user input may be read from datapre-stored in a look-up table, wherein the imaging parameter issubsequently applied to the image sensor 100. Alternatively, an inputvalue generated from user input may be input to a machine learningmodel, wherein a learning operation may be performed in advance and anresulting output applied to the image sensor 100 as the imagingparameter. In certain example embodiments, various forms of “featureinformation” (e.g., information regarding a particular type or shape ofobject being imaged, background information associated with at least aportion of a background for an image, composition information (e.g., thepositioning of an object with respect to a background), ambientenvironment illumination, imaging time, imaging location, contrastratios, color information (e.g., color temperature), etc.) may be inputto the machine learning model together with an input value generatedfrom user input.

FIG. 4 is a diagram further illustrating a portion of the pixel array110 of FIG. 3 according to an example embodiment.

Referring to FIG. 4, a pixel array PA of an image sensor in the exampleembodiment may include a plurality of pixels PX. The plurality of pixelsPX may be connected to a plurality of row lines ROW1 to ROWm (ROW) and aplurality of column lines COL1 to COLn (COL). The image sensor may drivethe plurality of pixels PX by a plurality of row lines ROW unit. As anexample, the time required for driving a selected driving line among theplurality of row lines ROW and reading out a reset voltage and a pixelvoltage from pixels PX connected to the selected driving line may bedefined as a single horizontal period. The image sensor may operate by arolling shutter method for sequentially exposing pixels PX connected tothe plurality of row lines ROW to light, respectively, or by a globalshutter method for simultaneously exposing pixels connected to all therow lines ROW to light, or another method.

A frame period FT of the image sensor may be defined as time periodsrequired for reading out a reset voltage and a pixel voltage from allthe pixels included in the pixel array PA. As an example, the frameperiod FT may be the same as or greater than a product of the number ofthe plurality of row lines ROW and a horizontal period. The shorter theframe period FT of the image sensor, the more the image sensor maygenerate image frames for the same period of time.

FIGS. 5 and 6 are circuit diagrams respectively illustrating possibleimplementations for a pixel that may be used in the pixel array 110 ofan image sensor according to an example embodiment.

Referring to FIG. 5, pixels included in an image sensor may include aphotodiode PD generating electric charges in response to light and aplurality of circuit devices for processing electric charges generatedby the photodiode PD and outputting an electrical signal. As an example,the plurality of circuit devices may include a reset transistor RX, adriver transistor DX, a select transistor SX, and a transfer transistorTX.

The reset transistor RX may be turned ON and turned OFF by a resetcontrol signal RG, and when the reset transistor RX is turned ON, avoltage of a floating diffusion FD may be reset to a power voltage VDD.When a voltage of the floating diffusion FD is reset, the selecttransistor SX may be turned ON by a select control signal SG and a resetvoltage may be output to a column line Col.

In an example embodiment, the photodiode PD may generate an electron ora hole as a main charge carrier in response to light. After the resetvoltage is output to the column line Col, and the transfer transistor TXis turned ON, electric charges generated by the photodiode PD beingexposed to light may move to the floating diffusion FD. As an example,the electric charge generated by the photodiode PD may be stored in acapacitor CFD present in the floating diffusion PD. The drivertransistor DX may operate as a source-follower amplifier amplifying avoltage of the floating diffusion FD, and when the select transistor SXis turned ON by the select control signal SG, a pixel voltagecorresponding to the electric charge generated by the photodiode PD maybe output to the column line Col.

Each of the reset voltage and the pixel voltage may be detected by asampling circuit connected to the column line Col. The sampling circuitmay include a plurality of samplers each including a first inputterminal and a second input terminal, and the sampler may receive a rampvoltage through the first input terminal. The sampler may compare a rampvoltage input through the first input terminal with a reset voltage anda pixel voltage input through the second input terminal. Ananalog-to-digital converter (ADC) may be connected to an output terminalof the sampler, and the analog-to-digital converter may output resetdata corresponding to a result of the comparison between the rampvoltage and the reset voltage, and pixel data corresponding to a resultof the comparison between the ramp voltage and the pixel voltage. Thecontrol logic may generate image data using a pixel signal correspondingto a difference between the reset data and the pixel data.

In the example embodiment illustrated in FIG. 6, the pixel may include aphotodiode PD, a reset transistor RX, a driver transistor DX, a selecttransistor SX, a transfer transistor TX, a switch device SW, and others.In the pixel circuit illustrated in FIG. 6, a conversion gain may beadjusted in response to turning on or turning off of the switch deviceSW.

The conversion gain of the pixel circuit may correspond to voltagechanges generated by an electric charge, and may be inverselyproportional to capacitance of a floating diffusion. In other words,when capacitance of the floating diffusion increases, a conversion gainof the pixel circuit may decrease. When capacitance of the floatingdiffusion decreases, a conversion gain of the pixel circuit mayincrease. Thus, a conversion gain may increase by turning on the switchdevice SW, and a conversion gain may decrease by turning off the switchdevice SW.

The conversion gain may affect performance of the image sensor. As anexample, when a conversion gain of the pixel circuit is set to conformto a low illumination environment, a pixel voltage generated in a highillumination environment may exceed a dynamic range of the image sensor,and accordingly, quality of an image may be deteriorated. When aconversion gain of the pixel circuit is set to conform to a lowillumination environment, the driver transistor DX may not sufficientlyoperate in a low illumination environment, and accordingly, a dark partof the image may not be sufficiently represented, or other issues mayoccur. In the example embodiment illustrated in FIG. 6, a conversiongain of the pixel circuit may be dynamically adjusted by turning on orturning off the switch device SW.

As an example, when the switch device SW is turned OFF, capacitance ofthe floating diffusion FD storing electric charges generated by thephotodiode PD may be determined as first capacitance CFD1. When theswitch device SW is turned ON, capacitance of the floating diffusion FDmay be determined to be a sum of the first capacitance CFD1 and secondcapacitance CFD2. In other words, by turning off the switch device SW,capacitance of the floating diffusion FD may decrease, and a conversiongain may increase, and by turning on the switch device SW, capacitanceof the floating diffusion FD may increase and a conversion gain maydecrease.

FIGS. 7, 8 and 9 are respective flowcharts illustrating methods ofoperating an electronic device according to various example embodiments.

Referring to FIGS. 2 and 7, a method of operating the electronic device10 in the example embodiment is assumed to begin by a command or userinput causing the execution of a camera application (S100). When thecamera application is executed, the electronic device 10 may senseambient illumination (e.g., the intensity of received incident light)(S110). The sensing of ambient illumination may be variouslyaccomplished using one or more camera(s) and/or a light sensorintegrated into the electronic device 10. In response to the sensedambient illumination, the processor 14 of the electronic device 10 maybe used to set a conversion gain for the image sensor 12 (S120). In oneexample embodiment, the electronic device may also collect various formsof feature information, such as color temperature information, ambientenvironment information, etc., while sensing the ambient illumination(S110).

In one approach, the conversion gain for the image sensor 12 may bedetermined in accordance with the capacitance of a floating diffusionregion. As an example, when the sensed ambient illumination (S110) isrelatively high, a conversion gain for the image sensor 12 may berelatively low, but when the sensed ambient illumination sensed (S110)is relatively low, a conversion gain for the image sensor 12 may berelatively high.

Once the conversion gain for the image sensor 12 is set (S120), theelectronic device 10 may determine an appropriate blur strength to beapplied to a result image (S130). Here, the electronic device 10 mayreference a look-up table providing pre-stored data associated with blurstrength. As an example, ambient illumination, conversion gain for theimage sensor 12, and blur strength determined in accordance with theambient illumination and conversion gain may be stored in a look-uptable stored in memory 13.

In an example embodiment, the look-up table may be setup to increaseconversion gain and thereby generate a result image with a relativelyshallow depth when ambient illumination is relatively low. In this case,random noise properties of the result image obtained in a lowillumination environment may improve. The look-up table may also besetup to decrease conversion gain and to thereby generate a result imagewith a deep depth when ambient illumination is relatively high. In thiscase, properties of a signal to noise ratio for the result imageobtained in a high illumination environment may improve. However, anexample embodiment thereof is not limited thereto, and the look-up tableof memory 13 may be used to store various relationships among ambientillumination, conversion gain, and blur strength using a variety ofapproaches in response one or more preferences indicated by user input.As an example, the look-up table may be setup for a result image to begenerated with a relatively shallow depth in a high illuminationenvironment, or a relatively deep depth in a low illuminationenvironment.

Referring to FIGS. 2 and 8, a method of operating the electronic device10 in the example embodiment is again assumed to begin with theexecution of a camera application (S200). When the camera application isexecuted, the electronic device 10 may display a preview image (S210) onthe display 11, and use the processor 14 to extract various featureinformation associated with the preview image (S220).

The electronic device 10 may further determine whether user input isgenerated with respect to the preview image (S230). As an example, auser may generate user input while the preview image is displayed on thedisplay 11 via a touch input capability provided by the display 11 ofthe electronic device 10 and/or manipulating mechanical keys provided bythe electronic device 10.

When it is determined that user input is generated (S230+YES), theelectronic device 10 may further determine a depth on the basis of thefeature information and the user input (S240). However, when it isdetermined that user input is not generated (S230=NO), the electronicdevice 10 may determine a depth based on the feature information (S250).However the depth is determined, a result image may be generated inresponse to an imaging function executed with respect to the previewimage (S260). Alternatively, in the example embodiment, blur strengthmay be determined using the feature information extracted from thepreview image, and the blur strength may be changed in accordance with auser input generated on the preview image. To improve user convenience,changes in blur strength, changing in accordance with the user, may bedisplayed on the preview image in real time.

In example embodiments, the electronic device 10 may change an actualaperture value of the camera. When it is possible to change an actualaperture value of the camera, the determined depth (S240) or (S250) maybe displayed on the result image according to an actual aperture valueof the camera. When the camera has a fixed aperture value, thedetermined depth (S240) or (S250) may be represented by increasing ordecreasing blur strength applied to a background of the result image byan image processing operation. As an example, a shallow depth may berepresented by increasing blur strength applied to a background, and adeep depth may be represented by decreasing blur strength applied to abackground.

The electronic device 10 in the example embodiment may provide an inputvalue generated in response to user input and/or feature informationextracted from the preview image to input nodes of a machine learningmodel, such that a learning operation may be performed to determine blurstrength using a value output to an output node of the machine learningmodel. The use of a machine learning model will be described in someadditional detail hereafter.

The electronic device 10 in the example embodiment may adaptivelydetermine a depth on the basis of user input generated with respect tothe preview image and/or feature information extracted from the previewimage. When a camera has a fixed aperture value, the depth may berepresented on the result image by adjusting blur strength. In anexample embodiment, a machine learning model, a learning operation ofwhich is performed in advance, may be used to determine the blurstrength, and accordingly, the result image based on an experience of auser may be provided to the user.

Referring to FIGS. 2 and 9, a method of operating an electronic device10 in again assumed to begin with execution of a camera application(S300). When the camera application is executed, the electronic device10 may display a preview image together with an aperture icon on thedisplay 11 (S310). In an example embodiment, the aperture icon may havea shape representing or depicting an analog aperture for a manuallyoperated camera.

As before, the electronic device 10 may be used to extract featureinformation from the preview image (S320). In the example embodiment,the electronic device 10 may determine blur strength applied to a resultimage on the basis of the feature information extracted (330) and mayrepresent the blur strength of the preview image in advance.

The electronic device 10 may adjust an opening area of an aperture iconin response to user input generated with respect to the preview image(S330). As an example, a user may provide a touch input to the display11 displaying the preview image and/or may adjust mechanical keysprovided by the electronic device 10.

For example, an opening area of the aperture icon may increase inresponse to a first user input generated on the preview image. Theopening area of the aperture icon may decrease in response to a seconduser input different from the first user input. As an example, the firstuser input may be a multi-touch input (or first finger gesture) forincreasing a distance (or gap) between at least two fingers or a slidinggesture indicating sliding in a first direction. The second user inputmay also be a multi-touch input (or second finger gesture different fromthe first finger gesture) for decreasing a distance between at least twofingers, or a sliding gesture indicating sliding in a second directionopposite to the first direction.

In this regard, it should be noted that “user input” may take manydifferent forms.

The electronic device 10 may adjust blur strength with respect to thepreview image displayed on the display 11 in response to an input valuegenerated by the user input and/or feature information extracted fromthe preview screen (S340). In the example embodiment illustrated in FIG.9, blur strength applied to a background of the preview image may changein real time by the user input received in the electronic device whilethe preview screen is executed. When an imaging operation is executed bya user, a result image may be generated with blur strength of when theimaging operation is executed (S350). Thus, a user may execute theimaging operation after checking blur strength of a background torepresent a depth desired by a user in advance, thereby obtaining adesired result image.

FIGS. 10 and 11 are respective diagrams illustrating a method ofoperating an electronic device according to an example embodiment.

In the example embodiment illustrated in FIGS. 10 and 11, a user mayexecute a camera application to image an object using an electronicdevice 200. The electronic device 200 may include a housing 210, adisplay 220, a front camera 231 and 232, and others, and the housing 210may include one or more input buttons 211 to 213. As an example, theinput buttons 211 to 213 may include a power button 211 and volumecontrol buttons 212 and 213.

When the camera application is executed, the display 220 may displayicons 221 to 223 illustrated in FIGS. 10 and 11. As an example, a firsticon 221 may be configured to intuitively represent blur strengthchanging in accordance with a user input. The first icon 221 may have ashape of an aperture which appears to physically open/close like amanually operated camera, consistent with the methods illustrated inFIGS. 7 and 8.

Generally, in a manually operated camera, a depth may be determined inaccordance with an aperture value of a camera, and blur strength of abackground of an object may be determined in accordance with a depth. Inan example embodiment, when user input for setting a depth to a shallowdepth is generated, the opening area of the first icon 221 having anaperture shape may increase, and blur strength of the background mayincrease. When user input for setting a depth to a deep depth isgenerated, the opening area of the first icon 221 having an apertureshape may decrease, and blur strength of the background may decrease.Thus, an interface in which the depth of the result image may becontrolled in the manner similar to the adjustment (opening or closing)of an aperture associated with a manually operated camera may beprovided.

A second icon 222 may be configured to indicate whether a smart imagingfunction is activated. As an example, a user may activate or deactivatea smart imaging function by applying a touch input to the second icon222. The smart imaging function may refer to an imaging functionassociated with certain example embodiments, and may be configured toprovide an optimized result image with reference to a pre-stored look-uptable, or the like, along with a user input corresponding to the secondicon 222. A third icon 223 for obtaining a result image in response to atouch input by a user may be displayed on one side of the display 220.

In an example embodiment, a user may change an aperture value (i.e., thedegree to which an aperture is opened or closed) by applying a firstfinger gesture 240 (e.g., selected from a group of possible fingergestures) to the display 220. Referring to FIG. 10, when a user inputsthe first gesture by increasing a distance (or gap) between two fingerswhile touching the display 220, an opening area of the first icon 221may increase. Thus, the opening area of the first icon 221 maycorrespond to an aperture value defined as an F value in a camera.

In contrast, when the user inputs a second gesture different from thefirst gesture by decreasing the distance (or gap) between two fingerswhile touching the display 220, the opening area of the first icon 221may decrease. Referring to FIG. 11, as the user applies the secondgesture, an opening area of the first icon 221 may decrease.

As will be appreciated by those skilled in the art, various fingergestures may be applied to a display as a touch input. Touch input is awidely used technique associated with various touch sensitive displays.

In an example embodiment, an opening area of the first icon 221 mayincrease or decrease in accordance with a user input, and at the sametime, blur strength applied to a background of a preview image and aresult image may also decrease or increase. As an example, as in theexample embodiment illustrated in FIG. 10, when a user input forincreasing an opening area of the first icon 221 is generated, blurstrength of a background included in a preview image may increase.Accordingly, when an imaging function is executed while an opening areaof the first icon 221 is increased, a result image with a strongerout-focusing effect may be generated. When a user input for decreasingan opening area of the first icon 221 is generated as in the exampleembodiment illustrated in FIG. 11, blur strength of the backgrounddisplayed on the preview image may decrease. When an imaging function isexecuted while an opening area of the first icon 221 is decreased, aresult image with a weaker out-focusing effect may be generated. Theblur strength may be changed through an image processing operationexecuted by an electronic device (1 or 10).

A user input for adjusting an opening area of the first icon 221 mayalso be implemented by a different gesture other than the gesture ofspreading or closing two fingers. As an example, when a user moveshis/her finger in a first direction along a boundary of the display 220,an opening area of the first icon 221 may increase, and when the usermoves his/her finger in a second direction opposite to the firstdirection, an opening area of the first icon 221 may decrease.

An aperture value may also be adjusted using at least a portion of theinput buttons 211 to 213 included in the display 220. For example, anaperture value may increase when a volume increase button 212 ispressed, and an aperture value may decrease when a volume decreasebutton 213 is pressed. The volume control buttons 212 and 213 and anincrease and a decrease of an aperture value may correspond to eachother in a converse manner, differently from the above-describedexample. By assigning the function of adjusting an aperture value to thevolume control buttons 212 and 213, an effect as if a user adjusts anaperture value while checking an overall preview image displayed on thedisplay 220 may be provided, and the user may check changes in depth,changing in accordance with the adjusted aperture values.

In example embodiments, the first icon 221 having an aperture shape maynot be provided on the preview image, or may be configured to havecertain transparency. Also, if a user input is not generated on thepreview image for a certain period of time, the first icon 221 maydisappear and may not be displayed. When a touch input is generated onthe preview image after the first icon 221 disappears, the first icon221 may be displayed again. Also, to improve convenience in the imagingoperation, a position of the first icon 221 displayed on the previewimage may be directly adjusted by a user.

FIG. 12 is a block diagram illustrating possible additional approachesto the operation of an electronic device according to an exampleembodiment.

Referring to FIG. 12, an electronic device 300 may include a neuralnetwork framework 310 and a memory 320 among other components. Theneural network framework 310 may be implemented in hardware, software,or combination of software and hardware. Images 321 taken by a user, aswell as a parameter database 322 associated with the adjustment ofimaging parameters for a camera and other elements, may be stored in thememory 320. In this regard, the memory 320 may take one of manydifferent forms including both discrete memory components and/ordistributed memory components.

In the illustrated example of FIG. 12, the parameter database 322includes a look-up table 323 and a library 324 among other components.The look-up table 323 may be used to store (e.g.,) certain featureinformation associated with one or more image(s), as well as an inputvalue corresponding to a user input. The combination of featureinformation and the input value may be used to determine blur strengthwhich in turn may be used to determine a depth of a result image, etc.In an example embodiment, the look-up table 323 may be received by theelectronic device 300 from a cloud 400 (e.g., a cloud basedcommunication connection). Once received by the electronic device 300,the look-up table 323 may be stored in memory 320. In this manner, theelectronic device 300 may receive new and/or updated data from the cloud400 at predetermined update intervals or in response to a user input.

As an example, communication capabilities related to the cloud 400 maybe provided by the electronic device 300, or a telecommunicationsservice linked to the electronic device 300. The cloud 400 may be usedto analyze images taken by one or more user once they are stored to thecloud 400. In this manner, a cloud-based database (DB) 410 may begenerated in relation to the stored images. As an example, processingcapabilities provided by the cloud 400 may classify the images and/orthe users into defined groups according to feature information, forexample. Feature information that may be used to classify images mayinclude, for example, object information, background information, timeinformation, location information, ages information, gender information,user information, etc. Whatever articular feature information is used,the cloud 400 may convert the feature information or aspects of thefeature information into (e.g.,) one or more blur strengths that may beapplied to image(s), image portion(s), filter(s), color(s), etc. Blurstrength may be determined and applied in the cloud using the database410, and may be selectively applied according to group(s).

When an update request for the look-up table 323 is received from theelectronic device 300, the cloud 400 may extract (e.g.,) dataappropriate to a user from the database 410 with reference to certainfeature information associated with the stored image(s) in order togenerate an updated (or new) look-up table 323. The updated look-uptable 323 may thereafter be transferred to and stored by the electronicdevice 300. And since the look-up table 323 may be updated in accordancewith indicated preferences and/or personalized feature informationassociated with a user (e.g., feature information previously indicatedby user input in relation to images) performance of cameraapplication(s) running on the electronic device 300 may improve.

The library 324 may be managed by the neural network framework 310, andblur strength determined by a machine learning model 312 as well asother parameters associated with camera operation may be included in thelibrary 324. Hereafter, an exemplary learning process will be describedin relation to the machine learning model 312 of the neural networkframework 310, the library 324 and other components.

The neural network framework 310 may include a parameter extractionmodule 311, the machine learning model 312, a manager module 313, amongother components. The parameter extraction module 311 may be used tocalculate input parameters and output parameters for the learningoperation of the machine learning model 312 from images 321 stored inthe memory 320. The images 321 stored in the memory 320 may beconfigured as result images taken and stored by a user. These images maybe added to or removed from the memory 320 according to user input.

As an example, the parameter extraction module 311 may obtain an inputvalue corresponding to a user input generated on a preview image when animage is taken, as well as certain feature information associated withat least a portion of the images 321 stored in the memory 320. As anexample, the input value may correspond to the user input generated onthe preview image to directly adjust blur strength after an image istaken, and may be represented in numeral value such as an aperture valueof a camera. And as previously noted, a great variety of featureinformation may also be referenced during this process. In anotherexample, the parameter extraction module 311 may obtain informationassociated with one or more blur strength(s) for one or more images 321stored in the memory 320.

The parameter extraction module 311 may apply (e.g., select and provide)certain feature information associated with selected images to inputnodes of the machine learning model 312 as input parameter(s). As anexample, the neural network framework 310 may control the machinelearning model 312 to perform a learning operation such that outputparameters provided at an output node of the machine learning model 312accurately corresponds to blur strength indicating a depth of therespective image. For example, the neural network framework 310 maycontrol the machine learning model 312 to perform a learning operationthat respectively adjusts the weighting of values included in hiddennodes of a hidden layer associated with the machine learning model 312.Once the learning operation of the machine learning model 312 iscomplete, the manager module 313 may store the input parameter(s),output parameter(s), etc. associated with the machine learning model 312in the library 324

As a user operates the electronic device 300, the catalog (e.g., agroup, list or number) of images 321 stored in the memory 320 may changeby the removal and/or addition of selected images. When it is confirmedthat a certain catalog of images 321 stored in the memory 320 haschanged sufficiently (i.e., has changed in relation to a particularvalue or number), the neural network framework 310 may control themachine learning model 312 to again perform the learning operation inorder to appropriately update the library 324. Alternatively, after acertain amount of time has elapsed since a previous performing of alearning operation of the machine learning model 312, the neural networkframework 310 may control the machine learning model 312 to againperform the learn operation and update the library 324.

In other words, in the example embodiment, the machine learning model312 may be controlled to perform one or more learning operation(s) withreference to one or more images 321 stored in the memory 320. Further,when a user activates the smart imaging function described in relationto FIGS. 10 and 11, certain feature information (e.g.,) featureinformation extracted from a preview image) as well as input value(s)associated with user input generated in relation to the preview imagemay be applied to the machine learning model 312. In this manner, theelectronic device 300 may provide a result image optimized according touser preference(s) with reference to an output of the machine learningmodel 312. As an example, the electronic device 300 may determine blurstrength for the result image with reference to the output of themachine learning model 312 in order to appropriately adjust anout-focusing effect.

As an example, it may be assumed that among the images 321 previouslytaken and stored by a user, certain images include human figure(s) asobjects against a background having a high illumination environment andshallow depth(s). As these depth(s) are relatively shallow, it may bedetermined that the user prefers a strong blur strength when an imageincludes a human figure and background characterized by a highillumination environment. Thus, when an object is a human figure and theassociated background is a high illumination environment, the machinelearning model 312 may be configured to perform the learning operationto output a strong blur strength resulting in a shallow depth. In thismanner, the electronic device 300 may generate a result image having ashallow depth by increasing blur strength applied to the background withreference to the output of the machine learning model 312.

In contrast, other images 311 stored in the memory 320 may include anumber of human figures as an object against a background having a lowillumination environment and therefore a relatively deep depth. Here,since the depth is relatively deep, it may be determined that the userprefers weak blur strength when several figures are included as anobjects in low illumination. When several human figures are recognizedin the preview image or it is determined that illumination is low, themachine learning model 312 may output a weak (or reduced) blur strength.The electronic device 300 may generate a result image with a deep depthby decreasing blur strength of the background with reference to theoutput of the machine learning model 312.

FIG. 13 is a chart illustrating a look-up table used in an electronicdevice according to an example embodiment.

Referring to FIGS. 12 and 13, the look-up table 323 stored in the memory320 of the electronic device 300 may be used to certain relationship(s)(or ratios) between illumination level (varying from dark to bright) andblur strength (varying from shallow to deep). The selection of theserelationships may be performed according to a selected group, category,or identified class of stored images. As an example, the electronicdevice 300 may generate a result image with a shallow depth as anaperture is open in low illumination environment, whereas the electronicdevice 300 may set to generate a result image with a deep depth as anaperture is closed in a high illumination environment. When a cameraapplication is executed by the electronic device 300, blur strength usedto obtain an optimal result image may be automatically set withreference to the illumination and blur strength relationship stored inthe look-up table 323. In this regard, it should be noted that as blurstrength changes, a Bokeh effect represented on the result image mayalso be vary.

Referring to an examples illustrated in FIGS. 12 and 13, blur strengthof an image sensor may be varied in accordance with illumination level,and blur strength may also be varied in accordance with parameters otherthan illumination level, where such other parameter(s) may be stored inthe look-up table 323. As an example, the depth of a result imagedetermined on the basis based of illumination level and conversion gainmay be stored in the look-up table 323.

Here, it should be noted that a conversion gain of the image sensor mayincrease as illumination is reduced, and decrease as illuminationincreases. Thus, by increasing conversion gain when illumination is low,a pixel signal having strength sufficient for generating an image in arelatively dark environment may be secured. Also, by decreasingconversion gain when illumination is high, the pixel signal will notbecome saturated. Thus, as conversion gain increases, a result imagehaving a shallow depth may be generated, and as conversion gaindecreases, the conversion gain and blur strength may be stored in thelook-up table 323 to generate a result image with a deep depth.

The look-up table 323 may be generated with reference to a plurality ofsample images. The look-up table 323 may be generated by collectingvarious aperture value(s), blur strength(s), and other featureinformation associated with the sample images. The look-up table 323 maybe stored in the cloud 410, accessed by the electronic device 300 asrequired, updated by processing capabilities in the cloud 410, etc.,such that the electronic device 300 may maintain an appropriatelyupdated look-up table 23 associated with sample images and revisedsample images over defined intervals or operating conditions.

In the example embodiment in FIG. 13, data used to generate a resultimage with an intermediate depth (e.g., chart elements P6, P7, and P8,for example) may be variously adjusted, (e.g.,) as tuning reference(s)and tuning direction for a constituent image sensor may be differentlymodified according to different manufacturers requirements. Nonetheless,the electronic device 300 may use the look-up table 323 to appropriatelydefine blur strength corresponding to intermediate illuminationenvironments, as well as intermediate conversion gain.

FIG. 14 is a flowchart summarizing in one example a method of operatingan electronic device according to an example embodiment.

Referring to FIGS. 12 and 14, a method of operating the electronicdevice 300 may begin by training a machine learning model 312 usingimages 311 stored in the memory 320 (S400). Here, the training of themachine learning model 312 may be performed, at least in part, byextracting feature information associated with the images 311 or imagingparameters respectively associated with the images 311. The extractedfeature information may then be applied as input parameters to themachine learning model 312 in order to determine a respective blurstrength for the images 311 as output parameters. This training of themachine learning model 312 may be executed, for example, by the neuralnetwork framework 310.

Once the machine learning model has been trained (S400), a cameraapplication may be executed by the electronic device 300 (S410), and apreview image may be displayed (S420). When the preview image isdisplayed, the electronic device 300 may extract feature informationassociated with the preview image and apply the extracted featureinformation to the machine learning model (S430). As an example, amethod of extracting the feature information from the preview image(S430) may be the same as a method of extracting the feature informationfrom each image stored in the memory 320 to perform the learningoperation of the machine learning model in the operation S400.

The electronic device 300 may determine blur strength using an output ofthe machine learning model 312 which has received the extracted featureinformation (S440). Since the camera application is executed after thetraining of the machine learning model 312 is complete, the blurstrength may be determined to correspond with the blur strength of animage that is the same as (or similar to) the preview image (440).Hence, the determined blur strength should be acceptable to the user,and a result image may be displayed in real time (S450). However, theuser yet generate another result image by checking the blur strengthrepresented on the result image by executing an imaging function.

Thus, if the blur strength determined by the machine learning model 312is unacceptable, the user may generate a new input with respect to thepreview image and adjust the blur strength accordingly.

FIG. 15 is a conceptual diagram further illustrating a method ofoperating an electronic device according to another example embodiment.

Referring to FIG. 15, an electronic device (e.g., like the electronicdevices shown in FIGS. 1A, 1B, 2, or 12) may extract feature informationfrom a plurality of regions (e.g., regions A1, A2, A3 and A4) indicatedin a preview image PRE. The number of the plurality of regions may varywith application and the electronic device may obtain featureinformation from the plurality of regions.

Referring to FIGS. 12, 14 and 15, certain feature information frompreviously stored images 311 stored in memory 320 may be input to themachine learning model 312 and a training operation performed (S400).The machine learning model 312 may then receive feature information andprovide blur strength appropriately required to generate a result imageoptimized to user preferences. The blur strength output by the machinelearning model 312 may be reflected in the preview image PRE in realtime. As an example, the blur strength may be blur strength applied tothe background or portions of the background of the preview image PRE.

Here, if the user does is not satisfied with the determined blurstrength, as now reflected in the preview image PRE, the blur strengthmay be further adjusted using user input such as a gesture input GES.For example, if the user determines that the blur strength reflected inthe preview image PRE is too weak, the user may increase the blurstrength using a gesture of spreading two or the user's fingers.

FIG. 16 is a conceptual diagram and FIG. 17 is a related chartillustrating a machine learning model (like the machine learning modelof FIG. 12) that may be used to determine one or more imagingparameter(s) in accordance with an aperture value in an electronicdevice according to an example embodiment.

Referring to FIG. 16, a machine learning model 500 may include an inputlayer 510, a hidden layer 520, and an output layer 530. The input layer510 may include a plurality of input nodes 511, the hidden layer 520 mayinclude a plurality of hidden nodes 521, and the output layer 530 mayinclude an output node 531. In an example embodiment, the output layer530 may include a single output node 531. In example embodiments, thenumber of the output nodes 531 may be varied.

A plurality of input parameters may be input to the plurality of inputnodes 511. The plurality of input parameters may correspond to featureinformation extracted from images previously taken by a user and storedin a memory of the electronic device. As an example, the plurality ofinput parameters may correspond to certain feature information likedescribed above. Here, an input value may correspond to user inputgenerated in relation to the preview image as well as featureinformation. That is, the input value may be determined by a user inputgenerated by a user in relation to the preview image to adjust blurstrength when an image is taken, and may be represented as an aperturevalue. The input value may not match an actual aperture value of acamera which has taken the respective image. For example, the user maygenerate an input on the preview image to set the strongest blurstrength when taking a second image, and to set the weakest blurstrength when taking a third image, and aperture values of the cameramay be the same when the second image and the third image are taken.

When the above-mentioned feature information may be extracted fromimages stored in the memory of the electronic device and arranged incertain order(s) prior to being input to the plurality of input nodes511, the hidden layer 520 of the machine learning model 500 may performa certain calculation using values input to the input nodes 511, therebyexporting an output value to the output node 531. As an example, thehidden layer 520 may perform the above-mentioned calculation by addingup overall values input to a plurality of hidden nodes 521,respectively, by transferring 1 or 0 to a subsequent node when a sum ofoverall values input to the plurality of hidden nodes 521, respectively,is greater than a certain threshold, or by applying a certain weightingvalue to a value transferred among the plurality of hidden nodes 521.

In an example embodiment, an output value output to the output node 531may correspond to blur strength representing a depth of the respectiveimage. When the output value output to the output node 531 does notmatch blur strength representing an actual depth of the respectiveimage, or a difference between the output value and the blur strengthexceeds a certain reference, the machine learning model 500 may adjustthe threshold value or the weighting value applied to the plurality ofhidden nodes 521. The machine learning model 500 may repeat the learningprocess for adjusting the threshold value or the weighting value untilthe output value matches the blur strength representing an actual depthof the respective image or a difference between the output value and theblur strength is less than a certain reference. As an example, theabove-described learning process may be performed on the images storedin the memory.

In the following description, a process of performing a learningoperation for the machine learning model 500 using a first image, asecond image, a third image and a fourth image stored in a memory of anelectronic device will be described with reference to FIGS. 16 and 17.Referring to FIG. 17, certain feature information may be used tocalculate one or more input parameter(s). For example, when inputparameter(s) calculated from the first image are input to the inputnodes 511 of the machine learning model 500, the learning operation ofthe machine learning model 500 may be performed to control an outputvalue of the output node 531 to be 8 which indicates blur strength ofthe first image. Similarly, the input parameter(s) calculated from thesecond image are input to the input nodes 511 of the machine learningmodel 500, and the learning operation of the machine learning model 500may be performed to control an output value of the output node 531 to be10 which indicates blur strength of the second image. The learningprocess of the machine learning model 500 may include a process ofadjusting weighting values applied to the hidden nodes 521, and thelike.

In other words, when the input parameters obtained from each imagestored in the electronic device are input to the machine learning model500, the learning operation of the machine learning model 500 may beperformed to control an output parameter of the machine learning model500 to correspond to blur strength of each image. Thus, when a cameraapplication is executed, and an aperture value, illumination, types ofan object and a background, a composition, and others, are determined,the machine learning model 500 may output blur strength for generating aresult image optimized to preference of a user.

In an example embodiment, when a camera application is executed, inputparameters extracted from the preview image may be input to the machinelearning model 500 having already been trained in order to determine anoptimized blur strength, and accordingly, a result image with a depthpreferred by a user may be provided. In example embodiments, the machinelearning model 500 may also output values other than the blur strength,such as a color, a contact ratio, and the like, for example.

FIG. 18 is a conceptual diagram further illustrating an operation of anelectronic device according to an example embodiment.

Referring to FIG. 18, an electronic device 600 includes a look-up table610 and a machine learning model 620 among other components.

When a camera application is executed, the electronic device maydetermine illumination 631. As an example, the illumination 631 may havea relatively high level outside on a clear day, for example. Incontrast, the illumination 631 may have a relatively low level indoorson a cloudy day or at night. One or more values indicating thedetermined illumination 631 may then be applied to the look-up table 610and the machine learning model 620. The look-up table 610 may output anappropriate blur strength 641 which determines a depth of a result imagewith reference to the illumination 631, and a conversion gain set inaccordance with the illumination 631 in an image sensor.

When a camera application is executed, a function of directly adjustingan aperture value 632 on a preview image may be provided to a user. Theinput value 632 corresponding to a user input generated on the previewimage may be represented as an aperture value. In some cases, an actualaperture value of the camera may not be changed by the input value 632.The input value 632 may be input to the machine learning model 620. Theelectronic device 600 may input feature information 633 obtained fromthe preview image to the machine learning model 620 in addition to theinput value 632. As an example, the feature information 633 may includeinformation on an object, information on a background, a composition ofan object and a background, and other information. The machine learningmodel 620 may receive the illumination 631, the input value 632, thefeature information 633, and other information, and may output blurstrength which determines a depth of the result image.

The camera application may determine a depth of a result image taken bya user with reference to the look-up table 610 and blur strengths 641and 642 output by the machine learning model 620. As an example, whenthe blur strengths 641 and 642 are relatively high, the cameraapplication may process a background of an object to be blurred and mayset a depth of a result image to be a shallow depth. When a plurality ofimage sensors are provided in the electronic device, a background may beprocessed to be blurred by calculating a depth of an object and abackground using a phase difference of a signal output by the imagesensors. When a single image sensor is provided, a background may beprocessed to be blurred by recognizing an outline of an object using asoftware algorithm.

In an example embodiment, the blur strength 641 output by the look-uptable 610 may be determined by the illumination 631. As an example, whenillumination is relatively low, a shallow depth may be applied, and blurstrength may decrease. When illumination is relatively high, a deepdepth may be applied, and blur strength may increase. However, anexample embodiment thereof is not limited thereto, and a relationshipbetween the blur strength 641 and the illumination 631 may be varied inaccordance with data of the look-up table 610.

The blur strength 641 output by the machine learning model 620 may bedetermined by the input value 632, the feature information 633, andother information, in addition to the illumination 631. The learningoperation of the machine learning model 620 may be performed on thebasis of images taken in the past by a user and stored. Accordingly, theblur strength output by the machine learning model 620 may bedynamically determined in accordance with preference of the user. In theexample embodiment, by using both of the blur strength 641 output by thelook-up table 610 provided by a manufacturer of the electronic deviceand the blur strength 642 output by the machine learning model 620, animage having an optimized quality may be provided to a user.

FIG. 19 is a flowchart summarizing a method of operating an electronicdevice according to an example embodiment.

Referring to FIG. 19, a method of operating an electronic device maybegin with executing a camera application (S500) and sensing an ambientillumination environment (S510). Once illumination is sensed, aconversion gain for a constituent image sensor may be set in accordancewith the illumination (S520). As described above, when ambientillumination is relatively high, a conversion gain may decrease, andwhen ambient illumination is relatively low, a conversion gain mayincrease. Also, the electronic device may determine blur strength(S530). The blur strength in the operation 5530 may be determined by thelook-up table stored in the electronic device. The look-up table mayinclude data determining a relationship among ambient illumination, aconversion gain, and blur strength.

The electronic device may apply the blur strength determined in theoperation S530 and may display a preview image (S540), and may extractfeature information from the preview image (S550). A user may check thepreview image in which the blur strength determined in the operationS530 is reflected, may determine whether to change the blur strength,and may generate a user input (S560). As an example, the user maygenerate a gesture input for increasing or decreasing the blur strengthon the preview image.

When the user input is generated in the operation S560, the electronicdevice may determine blur strength on the basis of the input valuecorresponding to the user input and the feature information which hasbeen extracted from the preview image (S570). When a user input is notgenerated in the operation S560, the electronic device may determineblur strength on the basis of the extracted feature information (S580).The blur strength determined in the operation S570 or the operation S580may be determined with reference to the blur strength of the operationS530. When a user executes an imaging function, a result image to whichthe blur strength determined in the operation S570 or the operation S580is applied may be generated (S590).

FIGS. 20, 21, 22 and 23 are various diagrams further illustratingoperation of an electronic device according to an example embodiment.

Referring to FIGS. 20 and 21, a preview image may be displayed on adisplay 720 in response to execution of a camera application of anelectronic device 700. Two objects 721 and 722 are present on thepreview image illustrated in FIG. 20, and the first object 721 may bedisposed more adjacent to the electronic device 700 than the secondobject 722. When the objects 721 and 722 present in different distancesfrom the electronic device 700 overlap each other and appear on thepreview image, blur strength may be set to be weak by a machine learningmodel provided in the electronic device 700, and both of the objects 721and 722 may be focused.

In the example embodiment illustrated in FIG. 21, the second object 722may not be present on the preview image differently from the exampleillustrated in FIG. 20. As only the single object 721 is present in theexample embodiment illustrated in FIG. 21, blur strength may be set tobe strong by the machine learning model provided in the electronicdevice 700. Accordingly, as illustrated in FIG. 21, the object 721 maybe focused, and a background 723 may be blurred such that a blur effectmay be strong. As described with reference to FIGS. 20 and 21, a smartimaging function may be provided by the machine learning model of theelectronic device 700.

In the example embodiment, an imaging parameter such as blur strengthmay be varied by simply changing the number and the positions of theobjects 721 and 722 under the same field of view, the same illumination,the same aperture value, and others. However, the example embodimentdescribed with reference to FIGS. 20 and 21 is merely an example, and anexample embodiment thereof may be varied depending on images used forthe learning operation of the machine learning model which provides thesmart imaging function.

Referring to FIG. 22, a preview image may be displayed on a display 820in response to execution of a camera application of an electronic device800. On the preview image illustrated in FIG. 15, a first icon 821having an aperture shape for intuitively displaying blur strengthapplied to a result image, a second icon 822 indicating whether a smartimaging function is activated, a third icon 823 corresponding to animaging button, and other elements, may be displayed.

The electronic device 800 of FIG. 22 may provide recommended settings824 received from a cloud to a user while the smart imaging function isactivated. The recommended settings 824 may be settings received from acloud server by the electronic device 800 with reference to featureinformation.

The recommendation settings 824 may be settings frequently selected byother users under conditions the same as or similar to the conditionsunder which the electronic device 800 obtains a current image, and mayinclude blur strength, a color filter, and the like, which affect aresult image. A user of the electronic device 800 may automatically setimaging parameters through a touch input 840 for selecting one of therecommendation settings 824, or may directly change blur strength andother element by generating an input for increasing or decreasing anopening area of the first icon 821 without consideration of therecommendation settings 824.

Referring to FIG. 23, a user may deactivate a smart imaging function ona preview image displayed on a display 920 of an electronic device 900.As an example, when the user inactivates the smart imaging function bytouching the second icon 922, an icon having an aperture shape fordisplaying blur strength applied to a result image may disappear,differently from the example embodiments described above. When the smartimaging function is inactivated, the blur strength may be maintained asa value obtained right before the smart imaging function is inactivated.Alternatively, the electronic device 900 may automatically set blurstrength with reference to the look-up table in which ambientillumination and blur strength determined in accordance with ambientillumination are stored.

As an example, when the smart imaging function is inactivated, a fourthicon 924 for adjusting blur strength may be additionally displayed asillustrated in FIG. 16. A user may adjust blur strength using a gestureinput 940 for moving a bar of the fourth icon 924, and the blur strengthmay be adjusted by software without necessarily using the machinelearning model stored in the electronic device 900.

According to the foregoing example embodiments, a preview image may bedisplayed in response to execution of a camera application, and a depthapplied to a result image may be determined (or set) on the basis ofvarious feature information, at least some of which may be extractedfrom the preview image. A depth applied to the result image may also oradditionally be determined (or set) on the basis of user inputassociated with the preview image. Blur strength of a backgroundincluded in the result image may be varied in accordance with the depthapplied to the result image. Accordingly, an electronic device capableof providing a result image better conforming to personal userpreferences may be obtained.

While the example embodiments have been shown and described above, itwill be apparent to those skilled in the art that modifications andvariations could be made without departing from the scope of theinventive concept as defined by the appended claims.

What is claimed is:
 1. A method of operating an electronic device, themethod comprising: displaying a preview image obtained by executing acamera application; extracting feature information from the previewimage; converting a user input into an input value in response to userinput generated in relation to the preview image, and setting a depthbased on the feature information and the input value; and generating aresult image in accordance with the depth by executing an imagingoperation.
 2. The method of claim 1, further comprising: displaying anaperture icon on the preview image; and adjusting an opening area of theaperture icon in response to the user input generated on the previewimage.
 3. The method of claim 2, wherein when the user input is touchinput indicating an increasing of distance between two touch regions,the opening area of the aperture icon is increased, and the depth is setto a shallow depth, and when the user input is a touch input indicatinga decreasing of distance between two touch regions, the opening area ofthe aperture icon is decreased, the depth is set to be deep depth. 4.The method of claim 2, wherein when the user input is touch inputindicating finger movement in a first direction, an opening area of theaperture icon is increased, and the depth is set to be shallow depth,and when the user input is a touch input indicating finger movement in asecond direction opposite to the first direction, an opening area of theaperture icon is decreased, and the depth is set to be deep depth. 5.The method of claim 1, wherein the feature information includes featureinformation extracted from the preview image.
 6. The method of claim 5,wherein the feature information includes at least one of objectinformation, background information, ambient environment information,and composition information.
 7. The method of claim 5, wherein thefeature information includes at least one of color temperatureinformation and an illumination level information.
 8. The method ofclaim 5, wherein the depth is determined based on an output value of amachine learning model by applying feature information as the inputvalue to the machine learning model.
 9. The method of claim 1, furthercomprising: setting the depth based on the feature information when userinput is not generated on the preview image before the imaging operationis executed.
 10. The method of claim 9, wherein the depth is determinedwith reference to a look-up table stored in a memory of the electronicdevice.
 11. The method of claim 10, wherein the depth is set to ashallow depth when a conversion gain of the image sensor increases, andthe depth is set to a deep depth when a conversion gain of the imagesensor decreases.
 12. The method of claim 1, wherein blur strengthassociated with a background of the preview image is determined inaccordance with the depth.
 13. A method of operating an electronicdevice, the method comprising: displaying an aperture icon having anaperture shape on a preview image in response to execution of a cameraapplication; adjusting an opening area of the aperture icon displayed onthe preview image in response to user input generated in relation to thepreview image; and adjusting blur strength associated with a backgroundof the preview image in response to user input.
 14. The method of claim13, wherein the opening area of the aperture icon and the blur strengthincrease when user input is a first gesture, and the opening area of theaperture icon and the blur strength decrease when user input is a secondgesture different from the first gesture.
 15. The method of claim 13,further comprising: executing an imaging operation in relation to thepreview image and applying the blur strength to generate a result image.16. The method of claim 13, wherein the blur strength is determinedbased on at least one of user input and feature information associatedwith the preview image.
 17. The method of claim 13, wherein the apertureicon is controlled to not be displayed when user input is not generatedfor a predetermined period of time on the preview image.
 18. The methodof claim 13, wherein the aperture icon is again displayed when userinput is generated on the preview image on which the aperture icon isnot displayed.
 19. A method of operating an electronic device, themethod comprising: performing a training operation for a machinelearning model by applying feature information respectively extractedfrom images stored in a memory as input parameters and applying blurstrength respectively extracted from the images as output parameters;extracting the feature information from a preview image in response toexecution of a camera application, and inputting the feature informationto the machine learning model; and determining blur strength to abackground of the preview image in response to an output value of themachine learning model.
 20. The method of claim 19, further comprising:changing the blur strength in response to user input generated inrelation to the preview image.